Generating virtual representations

ABSTRACT

Methods, computer systems and computer programs for generating a virtual representation of an interior space such as a room are provided. Techniques for capturing and normalizing sets of points for polygon mesh generation that utilize augmented reality toolkits are also described. The methods comprise: obtaining a first set of three-dimensional coordinates and at least one further set of three-dimensional coordinates, wherein the first set of three-dimensional coordinates comprises three-dimensional coordinates representing three-dimensional positions of points located on edges of walls of the interior space, and wherein each of the at least one further set of three-dimensional coordinates comprises three-dimensional coordinates representing positions of points located on edges of an extrusion in one of the walls of the interior space; and generating a polygon mesh representing the three-dimensional shape of the interior space, wherein generating the polygon mesh comprises: using the first set of three-dimensional coordinates to determine planes representing the walls of the interior space without considering any extrusions in the walls; and for each wall with one or more extrusions, using the respective determined plane and the respective one or more of the at least one further set of three-dimensional coordinates to determine a plurality of sub-meshes that in combination represent the respective wall excluding the respective one or more extrusions; and combining the plurality of sub-meshes into a mesh representing the wall with the one or more extrusions.

PRIORITY STATEMENT

This application claims priority under 35 U.S.C. § 119 to UK ApplicationNo. GB1807361.9 filed on May 4, 2018, the contents of which is herebyincorporated herein by reference in its entirety.

TECHNICAL FIELD

The invention relates to methods, computer programs and computer systemsfor generating virtual representations, in particular, virtualrepresentations of three dimensional interior spaces such as rooms.

BACKGROUND

Virtual representations of three dimensional objects and spaces may begenerated for various reasons. For example, virtual representations ofenvironments, buildings, objects and people may be generated for films,animation and gaming; virtual representations of anatomical objects maybe generated for medical imaging; and virtual representations ofbuildings, rooms and objects within buildings and rooms may be generatedfor architectural and interior design purposes.

Some techniques for generating virtual representations of objects andspaces involve the generation of a polygon mesh (sometimes called awireframe model), typically made up of triangles, that approximates the3D shape of the object or space for which the virtual representation isto be generated. The mesh is then input to a rendering engine which usestechniques such as shading and texture mapping to convert the mesh intoa virtual representation of the 3D object or environment for display ona screen. Rendering techniques and engines for converting a mesh into animage are well-known and will not be described in further detail.

Generating a polygon mesh for input to a rendering engine typicallyinvolves applying a mesh-generation technique to an array of predefinedvertices (three-dimensional coordinates of surface points of the objector space). According to some known polygonal modelling techniques:

-   -   An array of edges which connect pairs of the vertices is        generated (or may itself be predefined, in an edge table for        example);    -   Using the array of edges, all polygons (typically triangles),        which are closed sets of the edges, are generated;    -   All polygons on the same face plane are combined;    -   All polygon faces which are in the same horizontal or vertical        plane are grouped; and    -   All groups of polygon faces are combined to form the 3D        polygonal mesh of the object or space being modelled.

The predefined vertices that are used as an input to the mesh-generationalgorithm may be sourced from anywhere, but typically must be highlyaccurate if the mesh-generation algorithm is to produce a mesh thataccurately represents the shape of the 3D object. For generating meshesof buildings and interior spaces such as rooms of buildings, verticesare often captured using specialized equipment such as a laserrangefinder, operated by trained individuals. The complexity of thevertex capture process may therefore mean that mesh generation,particularly for interior spaces, is not accessible to untrained usersand is not amenable to real-time or near-real time applications.

SUMMARY OF THE INVENTION

Embodiments described herein address problems with known techniques forgenerating meshes that are used as inputs of a rendering engine, andprovide for the real-time generation of virtual representations ofinterior spaces such as rooms.

The inventors have appreciated that some known mesh generationtechniques, while effective, may be computationally demanding. This isespecially problematic for mobile devices such as smart phones andtablets, which have limited processing capabilities and battery life.Embodiments described herein provide for efficient mesh generation,which allows for the real-time or near-real-time generation of a virtualrepresentation of a space, including by mobile devices.

Further, the inventors have appreciated that existing vertex capturetechniques limit the accessibility of virtual generation of interiorspaces, and limits real-time or near-real time generation of virtualrepresentations of spaces. Embodiments described herein provide meshgeneration techniques which can make use of vertices captured withoutspecialized equipment and skills, and so permits all kinds of users togenerate virtual representations in real time or near-real time.Techniques for capturing vertices are also provided.

The scope of protection is defined in the independent claims to whichreference is now directed. Optional features are set out in thedependent claims.

According to a first aspect of the present invention, there is provideda method for generating a virtual representation of an interior spacesuch as a room. The method comprises obtaining a first set ofthree-dimensional coordinates and at least one further set ofthree-dimensional coordinates. The first set of three-dimensionalcoordinates comprises three-dimensional coordinates representingthree-dimensional positions of points located on edges of walls of theinterior space. Each of the at least one further set ofthree-dimensional coordinates comprises three-dimensional coordinatesrepresenting positions of points located on edges of an extrusion in oneof the walls of the interior space. The method further comprisesgenerating a polygon mesh representing the three-dimensional shape ofthe interior space. Generating the polygon mesh comprises using thefirst set of three-dimensional coordinates to determine planesrepresenting the walls of the interior space without considering anyextrusions in the walls; and for each wall with one or more extrusions,using the respective determined plane and the respective one or more ofthe at least one further set of three-dimensional coordinates todetermine a plurality of sub-meshes that in combination represent therespective wall excluding the respective one or more extrusions; andcombining the plurality of sub-meshes into a mesh representing the wallwith the one or more extrusions.

Generating a mesh that represents a very simple space which does nothave any extrusions such as doors, windows and fireplaces in its wallsmay be relatively straightforward. However, extrusions, which arepresent in most rooms, may vastly increase the complexity of some knownmesh generation techniques. This is because extrusions quickly increasethe number of three-dimensional coordinates/vertices required torepresent the space, such that the number of edges connecting verticesand the number of polygons connecting edges vastly increases.Additionally, vertices representing the extrusions can be encapsulatedwithin the edges representing the walls, which creates complex shapesfor which the calculation of polygons is also complex.

In contrast to these known techniques in which polygons are calculatedfor a complex shape which includes vertices for both the wall and theextrusions, the present invention stores and considers the array ofvertices which represent the walls of the interior space (without anyextrusions in the walls) and the arrays of vertices which represent theextrusions separately. This allows the complex shape to be separatedinto simple shapes, for which polygons can be efficiently calculated,before recombining the resulting meshes into a mesh representing thecomplex shape. Overall, this enables a more computationally efficientapproach to calculating polygon meshes of complex interior spaces, whichin turn allows for the real-time or near real-time generation of virtualspaces on mobile devices such as smart phone or tablet computers.

According to a second aspect of the present invention, there is provideda method for generating a virtual representation of an interior spacesuch as a room. The method comprises obtaining a first set ofthree-dimensional coordinates. The first set of three-dimensionalcoordinates comprises three-dimensional coordinates representingthree-dimensional positions of points located on edges of walls of theinterior space. The method further comprising generating a polygon meshrepresenting the three-dimensional shape of the interior space.Generating the polygon mesh comprises normalizing the three-dimensionalcoordinates of the first set of three-dimensional coordinates to accountfor capture drift; using the normalized first set of three-dimensionalcoordinates to determine planes representing the walls of the interiorspace; and using the determined planes representing the walls of theinterior space to determine polygon meshes representing the walls of theinterior space.

Normalization of the three-dimensional coordinates can also be appliedto the first and/or at least one further set of three-dimensionalcoordinates of the first aspect of the present invention.

Capture drift, which may occur if the calibration of the electronicdevice used to capture the three-dimensional coordinates drifts duringthe capture process, may result in improperly aligned planes, or planesthat do not accurately represent the interior space. Normalizing thecoordinates before further processing ensures that the planesrepresenting the walls of the interior space are properly aligned andform angles that accurately represent the actual interior space.

Normalizing the three-dimensional coordinates of the first set ofthree-dimensional coordinates and/or the at least one further set ofthree-dimensional coordinates may comprise comparing an angle betweentwo planes to a predetermined threshold angle, and adjusting at leastone three-dimensional coordinate if the angle passes the threshold. Theplanes may be planes representing walls or planes representing a ceilingor floor. The use of a threshold allows angles that are due to theactual shape of the interior space to be distinguished from angles thatexist in the obtained sets of three-dimensional coordinates due tocapture drift and/or inaccuracies in point capture.

The method may comprise, for each wall without any extrusions, using thecorresponding plane to determine a mesh representing the wall. In thisway, polygon meshes representing all walls of the interior space areobtained so that a virtual representation of the entire interior spacecan be generated.

The method may comprise providing one or more polygon meshes to arenderer for rendering the one or more polygon meshes, wherein each ofthe one or more polygon meshes represents the three-dimensional shape ofone or more walls of the interior space. In some cases, the method maycomprise combining all of the meshes representing all of the walls ofthe interior space to give a single mesh representing thethree-dimensional shape of the interior space, and providing the singlepolygon mesh representing the three-dimensional shape of the interiorspace to a renderer for rendering. Providing the renderer with a singlemesh may reduce processing and memory bandwidth requirements. In othercases, the method may comprise providing a plurality of groups ofpolygon meshes to the render, each group representing one or more wall.Providing the renderer with polygon meshes separately or in groups,rather than in combination, may allow re-rendering the mesh of one ormore walls without re-rendering the meshes of all other walls. Thisallows users to make changes to a wall, at the level of the mesh and/orrenderer, without having to perform computationally demanding renderingfor the entire interior space.

The method may comprise determining, for each of the at least onefurther set of three-dimensional coordinates, which of the determinedplanes the extrusion belongs to. Determining which of the determinedplanes the extrusion belongs to may comprise comparing the orientationof a plane through the points representing positions of points locatedon edges of the extrusion to the orientation of the determined planes.This allows the association between a wall and an extrusion to bedetermined without obtaining a single set of points that includes boththe wall and the extrusion, which as noted above increases thecomputational complexity of the mesh generation.

Using the respective determined plane and the respective one or more ofthe at least one further set of vertices to determine a plurality ofsub-meshes that in combination represent the respective wall excludingthe respective one or more extrusions may comprise translating orprojecting the respective extrusion onto the respective plane. Theextrusion may be parallelized to the plane prior to translating orprojecting the extrusion onto the plane, which may reduce the effects ofcapture drift and inaccuracies in point capture which can cause theextrusion to be improperly aligned with its wall plane.

Using the respective determined plane and the respective one or more ofthe at least one further set of three-dimensional coordinates todetermine a plurality of sub-meshes that in combination represent therespective wall excluding the respective one or more extrusions mayfurther comprise dividing the plane less the extrusion into a pluralityof sub-planes; and generating a sub-mesh for each sub-plane. In thisway, a complex shape that encapsulates an extrusion can be divided intosimple sub-planes (such as rectangles) for which mesh generation isparticularly straightforward. The sub-meshes generated form thesub-planes can then be combined to create a mesh for the wall.

Dividing the plane less the extrusion into a plurality of sub-planescomprises performing an extrapolation technique. For example, theextrapolation technique may comprise dissecting, for each of the one ormore extrusions, the plane along lines through a minimum and maximumextent of the extrusion. Extrapolation techniques may be particularlyefficient for extrusions with a regular polygon cross-section in thatthey may generate sub-planes with particularly simple shapes, for whichmesh-generation is particularly efficient.

Dividing the plane less the extrusion into a plurality of sub-planes maycomprise performing a splicing technique. For example, the splicingtechnique may comprise, for each of the one or more extrusions,dissecting the plane through a central point of the extrusion. Asplicing technique may be preferred to an extrapolation techniquebecause it can be applied to both regular and irregular extrusions.Further, a splicing technique generates relatively few sub-planes, whichreduces the number of sub-meshes that must be generated and subsequentlycombined.

The polygons of the polygon meshes may be triangles.

The first set of three-dimensional coordinates may comprise at least onethree-dimensional coordinate for each vertical edge, wherein a verticaledge is an edge where two adjacent walls of the interior space meet.Capturing points located on vertical edges, possibly without capturingany points on horizontal edges, provides for fast point capture whilestill allowing the determination of planes representing interior spaceswith complex wall configurations.

The first set of three-dimensional coordinates may comprise athree-dimensional coordinate for each horizontal edge, wherein ahorizontal edge is an edge where a wall of the interior space meets aceiling or floor of the interior space. Capturing points on horizontaledges, possibly in addition to points on vertical edges, allows interiorspaces with non-uniform floors and ceilings to be accurately captured.

The first set of three-dimensional coordinated may comprise a heightpoint indicating the height of the interior space. This may allow forthe accurate determination of wall planes without having to capturepoints on horizontal edges.

Obtaining the first set of three-dimensional coordinates may comprise:displaying, on a display of an electronic device, a live view of theinterior space as captured by a camera of the electronic device; and foreach of the edges, receiving a user input indicating a point on thedisplay corresponding to the edge; converting the user input into athree-dimensional coordinate; and storing the three-dimensionalcoordinate in memory of the electronic device. Likewise, obtaining eachof the at least one further set of three-dimensional coordinates maycomprises: displaying, on a display of an electronic device, a live viewof the interior space as captured by a camera of the electronic device;and for each of the extrusions, receiving user inputs indicating pointson the display corresponding to the edges of the extrusion; convertingthe user input into the three-dimensional coordinates; and storing thethree-dimensional coordinates in memory of the electronic device. Anaugmented reality toolkit of the electronic device may provide theability for three-dimensional interpretation of the live camera feed inorder to convert the user inputs into the three-dimensional coordinates.

Augmented Reality Toolkits such as ARKit included in Apple's (RegisteredTrade Mark) iOS 11 and ARCore included in Google's (Registered TradeMark) most recent version of the Android (Registered Trade Mark)operating system can provide the ability for three-dimensionalinterpretation of a live camera feed, such that three dimensionalcoordinates of points displayed on the screen of a device can bedetermined. This allows vertex capture to be performed quickly andwithout the use of specialized equipment and/or software that is notavailable to most users.

The user input may be converted into a three-dimensional coordinate andstored in ° memory as soon as the user input is received. Thissignificantly reduces the effect of capture drift. While subsequentnormalization of the three-dimensional coordinates is possible, it isdesirable to reduce the amount of capture drift in the first place.

Obtaining the first set of three-dimensional coordinates and/or the atleast one further set of three-dimensional coordinates may compriseretrieving a previously captured set of vertices from memory of anelectronic device.

A computer program, such as a mobile app, comprising instructions whichwhen executed by a computer cause the computer to perform the method forgenerating a virtual representation of an interior space such as a roomis also provided.

A non-transitory computer-readable medium storing instructions which,when executed by a computer, cause the computer to perform the methodfor generating a virtual representation of an interior space such as aroom is also provided. It will be understood that the non-transitorycomputer readable medium may be a medium such as, but not limited to, aCD, a DVD, a USB storage device, flash memory, a hard disk drive, ROM orRAM.

A computer system comprising one or more processors communicativelycoupled to memory and configured to perform the method for generating avirtual representation of an interior space such as a room is alsoprovided.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be further described by way of example only andwith reference to the accompanying figures in which:

FIG. 1 is a schematic diagram of an exemplary interior space with wallsand extrusions in the walls;

FIG. 2 is a flow diagram illustrating the processes involved ingenerating a virtual representation of an interior space;

FIG. 3 is a schematic diagram illustrating the positions of verticesthat are captured for the exemplary interior space of FIG. 1;

FIGS. 4A-4F are schematic diagrams illustrating how three dimensionalcoordinates may be captured using a mobile electronic device;

FIG. 5 is a flow diagram illustrating a process of generating a polygonmesh from sets of three-dimensional coordinates; and

FIGS. 6A-6H are schematic diagrams illustrating some of processes ofFIG. 5.

Like reference numbers are used for like elements throughout thedescription and figures.

DETAILED DESCRIPTION

By way of an example, FIG. 1 illustrates a three-dimensional interiorspace 10 for which a virtual representation may be generated.

The interior space 10 includes interior surfaces 111, 112, 113, 114, 121and 122. Interior surfaces 111-114 are walls of the interior space 10,whereas interior surface 121 is a ceiling and interior surface 122 is afloor of the interior space 10. In FIG. 1, walls 111 and 112 areopposite to each other, and walls 113 and 114 are opposite each otherwith wall 113 in the foreground and wall 114 in the background.

The interior surfaces of the interior space 10 have vertical edges (notnumbered) where two adjacent walls meet, and horizontal edges (also notnumbered) where a wall meets either the ceiling 121 or the floor 122.The interior space 10 also has corners (not numbered) where two adjacentwalls meet either the ceiling 121 or the floor 122. In this description,unless the context dictates otherwise, a point said to be “located on anedge” of an interior surface such as a wall 111-114, ceiling 121 andfloor 122 may refer to a point on a vertical edge (where two adjacentwalls meet), a point on a horizontal edge (where a wall and afloor/ceiling meet) or a corner point (where two adjacent walls andeither a ceiling or floor meet).

The interior space 10 illustrated in FIG. 1 has very simple interiorsurfaces 111-114, 121, 122, and it should be appreciated that a simpleinterior space 10 has been chosen for ease of illustration andexplanation. For example, while the interior space 10 has four walls at90 degrees to each other, interior spaces can have more or fewer walls,with the angle between adjacent walls being greater or less than 90degrees. Further, while the interior space 10 has a ceiling 121 with auniform height, such that each of the walls 111-114 has the same uniformheight, the ceiling 121 could have multiple different heights or besloped, such that the walls 111-114 could be of different or non-uniformheights.

It should also be appreciated that the terms “vertical” and “horizontal”are used in this description to differentiate between edges where twowalls meet and edges where a wall an either the ceiling or floor meet,and not to exclude edges that form an angle with the true vertical andtrue horizontal. Although the edges of the interior space 10 arevertical and horizontal, interior spaces with sloped walls, floors andceiling exist, but for the purposes of this description are described ashaving “vertical edges” and “horizontal edges”.

Returning to the interior space 10 of FIG. 1, some of the interiorsurfaces 111-114, 121, 122 of the interior space 10 have extrusions, 15,16 and 17. In particular, wall 111 has an extrusion 15 in the form adoor, and wall 112 has two extrusions 16, 17 in the form of two windows.

The term “extrusion” may refer to any feature of the interior space 10which projects from or into an interior surface 111-114, 121, 122 of theinterior space. Common examples of extrusions in interior spaces includewindows, doors and fireplaces, but others exist. The extrusions 15-17 ofFIG. 1 are rectangular, but extrusions can generally be of any shape,including regular polygons, irregular polygons, and shapes with curvededges. While the extrusions 15-17 are not shown as having any depth, itwill be appreciated that this is merely for ease of illustration: inpractice, extrusions project from or into an interior surface by atleast some amount. It should also be appreciated that while the term“extrusion” is sometimes used to refer to an object of a fixedcross-section, this need not be case for extrusions in interior surfacesof an interior space 10. However, in many cases extrusions such aswindows do have a uniform cross-section or at least a cross-section thatis uniform for much of the depth of the extrusion.

The extrusions 15-17 have edges, which in the case of the extrusions15-17 of FIG. 1 are vertical edges and horizontal edges (vertical andhorizontal relative to the edges of the interior surfaces 111-114, 121,122). However, the edges do not need to be horizontal or vertical: theycould be sloped relative to the edges of the interior surfaces 111-114,121, 122 or curved. The extrusions 15-17 of FIG. 1 also have cornerswhere two edges meet. In this description, unless the context dictatesotherwise, a point said to be “located on an edge” of an extrusion mayrefer to a point located on an edge which is straight or curved, and mayalso refer to a corner point where two or more edges meet.

The interior space 10 may be a room in a home, a room in a commercialspace or other type of building, or indeed any other kind of interiorspace. Generally, any space which is at least partially enclosed by oneor more interior surfaces may be considered to be an interior space forthe purposes of this description.

FIG. 2 illustrates a process 20 of generating a virtual representationof an interior space, such as the interior space 10 of FIG. 1.

Firstly, in step 21, measurements are made to capture three-dimensionalcoordinates of points in the interior space 10. These three-dimensionalcoordinates are stored for access by an electronic device. The pointsthat are captured, and techniques for capturing the points, aredescribed in more detail below with reference to FIGS. 3 and 4A-4F.

Next, in step 22, an electronic device, for example a mobile device suchas a smart phone or tablet computer, obtains the previously capturedpoints. The previously captured points may have been captured using theelectronic device itself, as discussed in more detail below withreference to FIGS. 3 and 4A-F. In this case, step 22 may take placetemporally immediately after step 21. Alternatively, the capture process21 may have taken place a more extended length of time before step 22,and may have taken place without the involvement of the electronicdevice that obtains the points.

Next, in step 23, the electronic device uses the obtained points togenerate a polygon mesh representing the interior space. The generationof the polygon mesh will be described in more detail below withreference to FIGS. 5-6.

Finally, in step 24, the polygon mesh generated in step 23 is rendered,converting the mesh into a virtual representation of the interior space.The rendering 24 may take place immediately after the mesh is generated,or may take place at a later time using a stored version of the polygonmesh. The mesh may be rendered by the same electronic device that itused to generate polygon mesh, or another electronic device.

Now turning to FIG. 3, this illustrates exemplary points that may becaptured in step 21 of the process 20 illustrated in FIG. 2, for theinterior space 10 illustrated in FIG. 1.

The captured points include a first set of points which, as will beexplained in more detail below, is used to generate planes representingthe interior surfaces of the interior space. The first set of pointscomprises, for each vertical edge of the walls 111-114 of the interiorspace 10, a point 11 a, 11 b, 11 c, 11 d located on the vertical edge.The points 11 a-d may be located anywhere along the length of theirrespective vertical edge, including the corners of the interior space 10(that is, where two adjacent walls 111-114 meet either the floor 122 orceiling 121). While embodiments described herein may only require onepoint per vertical edge, multiple points per vertical edge could also becaptured. However, having multiple points per vertical edge mayintroduce redundancy in the information that is required to generate theplanes representing the walls, and increase the amount of processingthat is involved in generating the polygon mesh. Embodiments describedherein aim to reduce both the amount of time taken to perform the pointcapture process and amount of processing required to generate a polygonmesh, so it may be preferable to limit the number of captured pointswhere possible.

The first set of points optionally further comprises a height point 12 alocated on the ceiling 121 of the interior space. Where the ceiling 121has multiple different heights, such that the walls 111-114 of theinterior space are not all the same height, multiple height points maybe captured. For example, one point per horizontal edge (where a wall111-114 meets the ceiling 121) may be captured if the ceiling does nothave a single height. Alternatively, a three-dimensional coordinate of apoint 12 a located on the ceiling 121 may not be captured at all.Instead, a default height or user-entered height may be used in steps 22and 23 of the process shown in FIG. 2, especially where the interiorspace 10 only has a single height.

The first set of points optionally further comprises a floor point (notshown) located on the floor 122 of the interior space 10. However, ifthe coordinates of the other captured points are defined relative to anabsolute zero that is located in the plane of the floor, no floor pointis captured. This is often the case, for example if the points arecaptured using a calibrated piece of the equipment. Where the floor 122is not level, one point per horizontal edge where a wall 111-114 meetsthe floor 122 may be captured.

It should be appreciated that due to the symmetry of the interior space10, a different first set of points could be obtained that containsinformation equivalent to the first set of points described above. Forexample, rather than obtaining a first set of points that includes aheight point 12 a and a point 11 a, 11 b, 11 c, 11 d located on eachvertical edge of the interior space 10, it would be possible to insteadobtain a point located on each horizontal edge of the interior space anda point located on either wall 111 or wall 112. Other possibilities willbe apparent to those skilled in the art. However, since many interiorspaces have uniform floors and ceilings as interior space 10 does, yetmany interior spaces have a wall configurations that are less uniformthan that of the interior space 10 of FIG. 1, it may generally bepreferable to capture points on vertical edges.

The obtained points also include at least one further set of points,each further set of points representing an extrusion in one of the wallsof the interior space. In the case of the interior space 10 of FIGS. 1and 3, there are three further sets of points: a set of points 15 a, 15b, 15 c, 15 d for the door extrusion 15 in wall 111; a set of points 16a, 16 b, 16 c, 16 d for the window extrusion 16 in wall 112; and a setof points 17 a, 17 b, 17 c, 17 d for the window extrusion 17 in the wall112.

The set of points for the door extrusion 15 in wall 111 includes a pointlocated at each corner 15 a, 15 b, 15 c, 15 d of the extrusion 15. Theset of points for the first window extrusion 16 in wall 112 includes apoint located at each corner 16 a, 16 b, 16 c, 16 d of the extrusion 16.The set of points for the second window extrusion 17 in wall 112includes a point located at each corner 17 a, 17 b, 17 c, 17 d of theextrusion 17.

While the captured extrusion points shown in FIG. 2 are located at thecorners of the respective extrusions, it should be appreciated that thisis not essential. For example, a point for each edge of an extrusion,located anywhere on the edge, could also be used, especially forrectangular extrusions such as those shown in FIG. 2, because thelocations of the corners of the extrusion may be inferred from theintersection of the edges. For more complex polygon extrusions, however,points located at each corner of the extrusion may be preferable. Forextrusions with curved edges, one or more additional points located onthe curved edge may be obtained in order to approximate the shape of thecurved edge.

Significantly, the sets of the three-dimensional coordinates describedabove for the interior space 10 are grouped separately and not combinedinto a single array comprising all of the points. That is, rather thanstoring a single array of vertices that includes all seventeen of thepoints shown in FIG. 2, a first group comprising the five wall points 11a-11 d and 12 a; a second group comprising the four door points 15 a-15d; a third group comprising the four window points 16 a-16 d; and afourth group comprising the four window points 17 a-17 d are stored. Aswill be explained in more detail below, grouping the points separatelyallows the mesh generation processor to utilize the groups of pointsseparately and thereby reduce the complexity of the mesh generationprocess.

The three-dimensional coordinates that are obtained for mesh generationmay have been captured in any one of a number of different ways,including using known techniques. For example, the coordinates of thepoints may have been captured using a laser rangefinder. However, inpreferred embodiments, the three-dimensional coordinates have beencaptured using an electronic device that utilizes an augmented realitytoolkit, which will now be described in more detail below with referenceto FIGS. 4A-4F.

FIG. 4A illustrates a mobile electronic device 40 such as a smart phoneor tablet computer. As is known, mobile electronic devices such asmobile electronic device 40 include various processors, memory, a touchdisplay 41 that allows the device 40 to receive input from a user, andat least one camera that captures images and can provide a live view ofwhat is being captured to the user via the display 41.

In addition to the one or more cameras, electronic devices such asdevice 40 typically include a range of sensors. For example, in additionto cameras, the electronic device 40 may include one or more of a GPStransceiver, an accelerometer, a gyroscope, a microphone, a compass, amagnetometer and a barometer. Electronic devices can use data capturedby such sensors to derive information about their surroundings and theirposition and movements within the surroundings.

The capabilities of some mobile electronic devices and their associatedoperating systems have recently been enhanced to provide so-calledaugmented reality (AR) toolkits. For example, Apple (RegisteredTrademark) has recently released iOS 11, which includes an AR toolkitcalled “ARKit”. Likewise, recent versions of the Android (RegisteredTrademark) operating system include an AR toolkit called “ARCore”. ARtoolkits such as ARKit and ARCore make use of the cameras and othersensors of mobile devices to deliver new functionality. For example, ARtoolkits may be able to analyse a scene captured by the camera to detectvertical and horizontal planes in a scene, and track the movement ofobjects and other features within a scene. Augmented Reality overlaysmay be displayed over a live view of the images captured by the camera,in order to supplement the functionality provided to the user.

One capability that can be provided using software implemented using anAR toolkit of a mobile electronic devices 40 is the determination ofthree-dimensional coordinates of points of interest in images capturedby the camera. When a user points the camera of their mobile electronicdevice 40 at a scene and is presented with a live view of the scene,they can indicate a point of interest in the scene by providing a touchinput to the screen, and software implemented using the AR toolkitdetermines the three-dimensional coordinates of the point.

In this regard, FIGS. 4B-4F illustrate a user interface of a softwareapplication which may be used to capture the three-dimensionalcoordinates in step 21 of the process 20 of FIG. 2.

Referring to FIG. 4B, when a user wishes to capture three dimensionalcoordinates of an interior space, they may first be presented with aninstruction 42 to calibrate their device 40. Calibrating the device 40typically involves moving the device 40 around the interior space. Theuser may acknowledge the instruction 42 by providing a touch input, forexample using the “GO” button 43, and then move their device 40 aroundto calibrate it.

During the calibration, the device 40 may determine one or morereference points or planes which it uses for future point capture. Forexample, the device 40 may identify a point, for example a point in theplane of the floor 122, and assign the point as the origin point or“absolute zero” with coordinates (0, 0, 0). The three-dimensionalcoordinates of all future captured points may then be relative to thisabsolute zero. During the calibration, the device 40 may also determineits own position and orientation relative to absolute zero.

Referring to FIG. 4C, once the device 40 has been calibrated, the useris presented with an instruction 44 to capture points corresponding tothe wall edges (points 11 a-d in FIG. 3). After acknowledging theinstruction 44 using the “GO” button 43, the user is presented with alive view of the images that are being captured by the camera. The userthen moves the device 40 towards an edge of a wall and points the cameraof the device 40 so that at least part of the edge is visible on thedisplay 41 of the device. The user then provides an input, such as atouch input, to indicate a point located on the edge. The electronicdevice captures the input, determines the three-dimensional coordinatecorresponding to the input point, and stores the three-dimensionalcoordinate.

As illustrated in FIG. 4D, the live view 45 of the images captured bythe camera that are presented to the user may be overlaid with an ARoverlay such as a pin 46 to show the user the locations of the pointsthey have input. If the user mistakenly drops a pin 46, or considersthat the pin 46 has not been accurately placed at a relevant point, theymay be able to remove and, if necessary, replace the pin 46. The usermay also be presented with other AR overlays to help them capture therelevant points. For example, the user may be presented with an ARoverlay that prompts them to move the device closer to the edge, or tiltor otherwise reorient the device so as to improve the capture of thepoint.

The user repeats this process for all of the edges to capture all of therelevant points. As explained above with reference to FIG. 2, this mayinclude points for all of the vertical edges and/or all of thehorizontal edges. Where the ceiling 121 has a single height, the usermay not need to capture, for example, horizontal edges, and may insteadcapture or input a single height point such as height point 12 a. Thecapture/input of the height point may take place as part of the walledge capture process, the calibration step, or in an entirely separatestep.

As explained previously, all of the wall edge points are stored inassociation with one another as a first set of points. In this way,during mesh generation, an electronic device is able to retrieve andprocess the first set of points without also having to process thepoints of any of the extrusions. As will be described in more detailbelow with reference to FIGS. 5-6, this reduces the complexity of themesh-generation processing.

Having captured the wall points, the user is now required to capture theextrusion points. This is described below with reference to FIGS. 4E-4F.

Turning to FIG. 4E, once the user has confirmed they have completed thewall point capture process, they are presented with an instruction 47 tocapture points corresponding to the door edges (points 15 a-d in FIG.3). After acknowledging the instruction 44 using the “GO” button 43, theuser is presented with a live view of the images that are being capturedby the camera. The user then moves the device 40 towards an edge of thedoor 15 so that at least part of the edge is visible on the display 41of the device. The user then provides an input, such as a touch input,to a location on the edge to drop a pin. The electronic device capturesthe input, determines the three-dimensional coordinate corresponding tothe input, and stores the three-dimensional coordinate. The user thenrepeats this until all of the door points 15 a-d have been captured.

As explained previously, the door points 15 a-d are stored inassociation with each other as a set of points. If there are multipledoors in the interior space, each door has a separate set of points. Inthis way, an electronic device performing mesh-generation processing isable to process each door separately from the walls and otherextrusions.

Turning to FIG. 4F, once the user has confirmed they have completed thedoor point capture process, they are presented with an instruction 48 tocapture points corresponding to the edges of a window. As there are twowindows 16, 17 for the interior space 10 of FIG. 1, the user may firstcapture points for one of the windows (points 16 a-d of window 16),indicate when they have finished capturing points for the first window16, and then capture points for the other window (points 17 a-d ofwindow 17).

As with the wall point capture process of FIGS. 4B-4C and the door pointcapture process of FIG. 4D, after acknowledging the instruction 48 usingthe “GO” button 43, the user is presented with a live view of the imagesthat are being captured by camera. The user then moves the device 40towards an edge of a window so that at least part of the edge is visibleon the display 41 of the device. The user then provides an input, suchas a touch input, to a location on the edge to drop a pin. Theelectronic device captures the input, determines the three-dimensionalcoordinate corresponding to the input, and stores the three-dimensionalcoordinate. The user then repeats this until all of the first set ofwindow points 16 a-d have been captured. The process is then repeatedfor the second set of window points 17 a-d.

As will be appreciated from the previous explanation, the first set ofwindow points 16 a-d will be stored as a separate set of points, and thesecond set of window points 17 a-d will be stored as a separate set ofpoints. This allows each window to be processed separately from theother windows, extrusions and walls.

It will be appreciated that since different interior spaces havedifferent numbers and different types of extrusions in the walls, theprocess described above with respect to FIGS. 4B-4F can be adaptedaccording to the interior space. For example, before of after thecalibration step illustrated in FIG. 4B, the user may be prompted tospecify the types and number of extrusions that are present in theinterior space. The user can then be presented with point captureinstructions in accordance with the types and numbers of extrusions theyhave specified. As another example, for each different type of extrusionthat is recognised by the software application (doors, windows andfireplaces for example), the user may be repeatedly asked to capture anextrusion of a particular type until they confirm that they havecaptured all of the extrusions of that type. For example, the user mayfirst be presented with an instruction to capture door edges. Havingcaptured a first set of door edges, the user will again be asked tocapture door edges. If there are no doors left to capture, the user maybe able to use the touch display 41 to select that there are no moredoors to capture. The user may then be asked to capture points for anextrusion of the next type.

Advantageously, each three-dimensional coordinate is stored as soon asthe corresponding user input has been received and converted into athree-dimensional coordinate. That is, rather than waiting for allpoints, or all points in a given set of points, to be captured beforeconversion and storage, conversion and storage takes place immediatelyafter an input is received. This helps reduce the impact of so-calledcapture drift. Capture drift can arise due to the loss in calibrationover time, for example due to the mobile electronic device 40effectively losing its position and orientation within the space, whichit established during the calibration step. Capture drift increases overtime, especially following sudden changes in device position andorientation, so converting an input and storing the resulting coordinateas the input is captured reduces the amount of capture drift associatedwith each three-dimensional coordinate.

While the points can be captured using any point capture process, itwill be appreciated that the point capture process described above withreference to FIGS. 4A-4F has the advantage that it does not require anyspecialized equipment or any specialist skills. While thethree-dimensional coordinates captured using this process may berelatively inaccurate compared to, for example, points captured by atrained individual using a laser rangefinder, the separation of thepoints into separate sets, along with the normalization andparallelization techniques described below with reference to FIG. 5,allow the mesh generation techniques described below to create accuratevirtual representations of the interior space.

After having completed the point capture process of FIGS. 4A-4F, oranother point capture process, a first set of points representing walledges and at least one further set of points representing extrusions areavailable for use by an electronic device in order to generate a meshrepresenting the interior space. Mesh generation will now be describedwith reference FIGS. 5-6.

FIG. 5 illustrates the mesh generation process 23 of FIG. 2 in moredetail.

Firstly, in step 231, an electronic device such as electronic device 40normalizes the first set of coordinates obtained in step 22 of process20 to account for capture drift.

As explained above, in order to reduce the effect of capture drift, userinputs are preferably converted to three-dimensional coordinates andstored as soon as the user inputs are received. However, there is stilllikely to be some capture drift in the obtained three-dimensionalcoordinates, especially for the later-captured points, so an initialnormalization of the first set of coordinates is preferably performed toimprove the accuracy of the polygon mesh that will be generated from thepoints.

Normalizing the first set of three-dimensional coordinates involvescomparing the x-, y- and z-coordinate values of the points and makingadjustments to the values to create a set of coordinates that moreaccurately describe the walls, ceiling and floor of the interior space.Such a set of points should be internally consistent, andwell-constrained given the constraints of the interior space. Forexample, it will be appreciated that each point representing a verticaledge should lie in two different planes of the interior space (that is,an edge point should lie in the planes of two adjacent walls, where thewall planes intersect). However, capture drift may mean that some of thecaptured coordinates are not accurate, and that the requirement thateach point lies in two planes cannot be met at the same time for eachand every one of the captured wall edge points. As another example,capture drift and/or inaccuracies in point capture may mean that thereis angle between adjacent walls, or a ceiling or floor is sloped, evenfor interior spaces where the walls are actually perpendicular and/orthe ceiling flat. The normalization process makes adjustments to thecoordinate values to account for capture drift and other inaccuracies.

An example of a normalization process is illustrated in FIGS. 6A and 6B.

FIG. 6A illustrates the normalization of points on vertical edges. Inthis example, points lying on three vertical edges 50, 51 and 52 havebeen captured during the point capture process 21. Assuming verticalplanes through these points, the vertical edges 50, 51, 52 define twoadjacent walls which form an angle, x, between them. As part of thenormalization process 231, the angle x may be compared to a predefinedthreshold/tolerance and the coordinates of one or more points of thevertical edges 50, 51, 52 adjusted if the angle x passes thethreshold/tolerance.

For example, if the difference between the angle x and 90 degrees isless than the predefined tolerance (5 degrees, for example), it may beassumed that the two walls are actually perpendicular to each other andthat the difference in angle is due to capture drift and/or inaccuratepoint capture. In this case the coordinates of corner point 53 may beadjusted. In particular, the coordinates of points 53 may be adjusted tothose of point 54, which results in the angle between the two wallplanes being 90 degrees.

Alternatively, if the difference between the angle x and 90 degrees isgreater than the predefined tolerance, it may be assumed that thereactually is an angle between the two walls, because a difference greaterthan the tolerance is unlikely to be solely a result of capture driftand/or inaccurate point capture. In this case, the coordinates of thepoints may not be adjusted.

Referring now to FIG. 6B, this illustrates the normalization of pointson horizontal edges. In this example, points on horizontal edges 55 and56 have been captured during the point capture process 21. Assuming theedges 55 and 56 intersect at a point 53 on the vertical edge 51, theedge 56 forms an angle, y, with the horizontal. That is, the edge 56 asdefined by the captured point results in a sloped ceiling with an angley to the horizontal. As part of the normalization process 231, the angley may be compared to a predefined threshold/tolerance and thecoordinates of one or more points of the horizontal edges 55, 56adjusted if the angle y passes the threshold/tolerance.

For example, if the angle y is less than the predefined tolerance (10degrees, for example), it may be assumed that the ceiling is actuallynot sloped, and that the angle y is due to capture drift and/orinaccurate point capture. In this case the coordinates of corner point53 may be adjusted. In particular, the coordinates of points 57 may beadjusted to those of point 58, which results in a truly horizontal edgethat defines a flat ceiling.

Alternatively, if the angle y is greater than the predefined tolerance,it may be assumed that the ceiling actually is sloped, because adifference greater than the tolerance unlikely to be solely a result ofcapture drift and/or inaccurate point capture. In this case, thecoordinates of the points may not be adjusted.

The predefined thresholds/tolerances described above may vary dependingon the AR toolkit being used. For example, for AR toolkits thatexperience relatively little capture drift and/or inaccuracies, thetolerances may be reduced. Other factors may make it preferable toadjust the tolerance. For example, point capture tends to be lessaccurate for points lying on horizontal edges, as users may not be ableto get as close to a horizontal edge as a vertical edge because somehorizontal edges are at ceiling-level. A higher threshold/tolerance maytherefore be used for the normalization of points on horizontal edges.

The normalization process 231 may also be applied to the at least onefurther set of coordinates representing the extrusions, to ensure thatthe points representing an extrusion lie in a common plane. However, asthe extrusions may be parallelized to their respective wall planes instep 234 described below, it may not be necessary to normalize the atleast one further set of coordinates.

Next, in step 232, the first set of coordinates is used to determinepoint arrays that define the planes that represent the walls of theinterior space. That is, points located at the extreme corners of theplanes representing the walls (i.e. the points where the wall meets theceiling or floor) are determined.

To illustrate this, with reference to FIGS. 1 and 3, two edge pointshave been captured for the wall 114 of the interior space 10: points 11b and 11 c. Utilizing these two edge points 11 b and 11 c, and alsoutilizing absolute zero and the height point 12 a, the four cornerpoints located at the extreme corners of the wall plane can bedetermined. For example, the three-dimensional coordinate (x, y, z) ofthe top left corner of the wall 114 can be determined using the x- andy-coordinates of the point 11 b and the z-coordinate of the height point12 a. The three-dimensional coordinate of the bottom left corner of thewall 114 can be determined using the x- and y-coordinates of the point11 b and the z-coordinate of absolute zero. The three-dimensionalcoordinate of the top right corner of the wall 114 can be determinedusing the x- and y-coordinates of the point 11 a and the z-coordinate ofthe height point 12 a. The three-dimensional coordinate of the bottomright corner of the wall 114 can be determined using the x- andy-coordinates of the point 11 a and the z-coordinate of absolute zero.Turning to FIGS. 6C and 6D, described in more detail below, it can beseen that there are four corner points A1-A4 for a wall. Such cornerpoints are the output of step 232.

Next, in step 233, the correspondence between the walls and theextrusions is determined. That is, it is determined which extrusion(defined by its associated set of points) belongs to which wall/plane.

In one example, in order to determine which wall a given extrusionbelongs to, an angle between the plane in which the extrusion lies (asdefined the set of points representing the given extrusion) and a wallplane is determined. If the angle between the plane of the extrusion andthe plane of the wall is small, for example less than a threshold suchas 5 degrees, the extrusion is determined to belong to that wall. If theangle is above the threshold, the angle is calculated for another wallplane and this is repeated until the angle is below the threshold. If nocalculated angle is below the threshold, the wall plane which generatedthe smallest calculated angle may be chosen.

It will be appreciated that each extrusion should lie in the plane ofits associated wall, so if there are no inaccuracies in the pointcapture process, the angle should be zero for the associated wall.However, due to inaccuracies in the capture process and due to capturedrift, the calculated angle will not typically be zero, so a thresholdis used. The threshold that is used can be varied. For example, if theadjacent walls of the interior space are expected to be perpendicular, alarger threshold can be used. This is because the possibility of amistaken determination will only arise if the smallest angle between awall and an extrusion is approaching about 45 degrees, and it isunlikely that inaccuracies in the capture process would be sosignificant that they would result in such a large angle. On the otherhand, if the angle between adjacent walls could be quite shallow, asmaller threshold would be appropriate as otherwise a mistakendetermination could be made. In general, the threshold angle should besmaller than the shallowest angle between adjacent walls of the interiorspace.

It will also be appreciated that in many interior spaces (includinginterior space 10 of FIGS. 1 and 3), opposite walls are parallel orclose to parallel, so a small angle may be determined for two differentwalls. Therefore, to avoid mistaken determinations based on the anglebetween the extrusion and the wall plane, a distance (such as a distancebetween the centres of the wall plane and extrusion) may be used in thedetermination of the correspondence. If there is a large distancebetween the extrusion and wall plane, the extrusion will not belong tothat wall, even if the angle between the extrusion and wall is small.

Next, in step 234, for each extrusion, the extrusion is parallelized toand projected to its corresponding wall plane. This is illustrated inFIG. 6C, in which an extrusion represented by extrusion corner pointsB1-B4 is parallelized onto and projected or translated onto the wallplane represented by the wall corner points A1-A4.

If there were no inaccuracies in the point capture process, theextrusions would already be parallel to their respective wall planes.However, inaccuracies and capture drift mean that this may not be thecase. The x-, y- and z-coordinates of the points representing theextrusions may therefore be analysed and adjusted so that the planesdefined by the extrusions are parallel to their respective planes. It isnoted that the extrusions are parallelized to the wall planes, and notvice versa. Projecting the wall planes onto the planes of the extrusionscould result in a set of wall planes in which some of the adjacent wallsdo not share a common edge, and do not together create a closed set ofwalls.

Having parallelized the extrusions onto their corresponding wall planes,all of the extrusions belonging to a given wall are projected ortranslated onto said wall, as illustrated in FIG. 6C. The result, anexample of which is shown in FIG. 6D, is a wall plane (defined by pointsA1-A4) containing void(s) corresponding to its extrusion(s), as can beseen in FIG. 6D. A more complex example of a wall plane containing twovoids 18, 19 is shown in FIG. 6E.

Now turning to step 235 in FIG. 5, the part of the plane illustrated inFIGS. 6D and 6E that does not include the projected extrusion(s) isdivided into a plurality of sub-planes. That is, the complex shape thatencloses/encapsulates the extrusions/voids (a single extrusion/voidB1-B4 in FIG. 6D and two extrusions 18, 19 in FIG. 6E) is divided into aplurality of simple shapes that do not enclose/encapsulate theextrusion/void.

Dividing the wall plane, less the extrusions, into sub-planes mayinvolve performing an extrapolation technique based on the minima andmaxima points of the extrusions. An extrapolation technique isillustrated in FIG. 6D. Alternatively, it may involve a splicingtechnique that dissects the plane based upon the central point of theextrusion/void. A splicing technique is illustrated in FIGS. 6E-6G.

FIG. 6D illustrates an extrapolation technique which divides the planeless the void into four rectangular sub-planes 61, 62, 63, 64 thatindividually do not encapsulate the extrusion. In particular, in FIG.6B, the horizontal minima are represented by points B1 and B4, and thehorizontal maxima are represented by points B2 and B3. A vertical linethrough the horizontal minima points B1 and B4 is extrapolated to whereit intersects the wall plane, at points C1 and C4. Likewise, a verticalline through the horizontal maxima points B2 and B3 is extrapolated towhere it intersects the wall plane, at points C2 and C3. This creates afirst rectangular sub-plane 61 defined by points A1, C1, C4 and A4; asecond rectangular sub-plane 62 defined by points C1, C2, B2 and B1; athird rectangular sub-plane 63 defined by points C2, A2, A3 and C3, anda fourth rectangular sub-plane 64 defined by points B4, B3, C3 and C4.

It will be appreciated that the extrapolation technique described abovecould equally be applied using horizontal lines through the verticalminima B3 and B4 and the vertical maxima B1 and B2. The result would befour different, but essentially equivalent, rectangular sub-planes.

It will also be appreciated that the technique could be applied to awall plane with multiple voids, such as for wall 112 of interior space10 in FIG. 1. Performing the extrapolation technique for the wall 112could, for example, divide the wall plane into five rectangularsub-planes.

FIGS. 6E-6G illustrate an exemplary splicing technique for dividing awall plane 70, less its extrusions/voids 18, 19, into a plurality ofsub-planes 71, 721, 722 that individually do not encapsulate theextrusions 18, 19.

According to the splicing technique, referring to FIG. 6E, the left-mostextrusion 18 in the wall 70 is considered first. A vertical line throughthe centre of the extrusion 18 is extrapolated to where it intersectsthe wall plane 70, resulting in two regions of the wall plane: a firstregion 71 to the left of the centre of the extrusion 18 and a secondregion 72 to the right of the centre of the extrusion 18. Since theleft-most extrusion 18 has been considered first, the region 71 to theleft the extrusion 18 does not encapsulate any extrusion or void, and istherefore considered to be a first sub-plane 71. If the wall plane 70only had one extrusion, the second region 72 to the right of the centreof the extrusion 18 would also not encapsulate a void, so could be usedas a sub-plane. However, since the wall plane 70 includes a secondextrusion 19, the second region 72 to the right of the left-mostextrusion 18 encapsulates extrusion 19, so a further splicing step isrequired.

Now referring to FIG. 6F, a vertical line through the centre of theother extrusion 19 is extrapolated to where it intersects the wall plane70. This divides the second region 72 of FIG. 6E into two regions: aregion 721 that is to left of the centre of the extrusion 19 (and to theright of the centre of the extrusion 18) and a region 722 that is toright of the extrusion 19. Since neither of the regions 721, 722encapsulate any extrusion, they are considered to be second and thirdsub-planes 721, 722 of the wall plane 70.

Finally, referring to FIG. 6G, this illustrates the wall plane 70 in thewider context of a 3D interior space.

It will be appreciated that if the wall plane 70 included furtherextrusions, the above process would be repeated until no regionsencapsulate an extrusion. It will also be appreciated that modificationscould be made to the splicing technique. For example, the splicingtechnique could start with the most-right extrusion 19 rather than themost-left extrusion 18. Also, horizontal lines through the centres ofthe extrusions could be used instead of vertical lines.

In the example of FIGS. 6E-6G, this would have resulted in twosub-planes rather than three, though it will be appreciated that the twosub-planes would have been of a more complex shape.

Having divided the plane, less the extrusion(s) into the sub-planes, instep 236 of FIG. 5, sub-meshes are generated from the sub-planes. Thisinvolves performing a polygon generation process on the sub-planes,preferably a polygon triangulation process which generates trianglesfrom the sub-planes.

Referring again to FIG. 6D, the triangles generated from the sub-planesare shown in dotted lines. A sub-mesh consisting of two triangles isgenerated from the first sub-plane 61 by joining points A4 and C1. Asub-mesh consisting of two triangles is generated from the secondsub-plane 62 by joining points B2 and C1. A sub-mesh consisting of twotriangles is generated from the third sub-plane 63 by joining points A2and C3. A sub-mesh consisting of two triangles is generated from thefourth sub-plane 64 by joining points B4 and C3. It will be understoodthat other triangles could also be generated.

It will be appreciated that since the sub-planes 61-64 have simpleshapes, the generation of the sub-meshes in step 236 is relativelysimple and computationally efficient. In contrast, had a polygongeneration process been performed on a single set of points comprisingpoints A1-A4 and B1-B4, the connection of vertices and edges to formpolygons would have been more complex and computationally demanding. Byobtaining and utilizing the wall points and extrusion points separately,so as to create simple shapes for the mesh generation step 236, theexamples described herein save on processing and can be performed inreal-time.

Referring now to FIGS. 6E-6G, the triangles generated from thesub-planes 71, 721, 722 are shown in dotted lines. It will beappreciated that since the sub-planes 71, 721, 722 have a more complexshape than the sub-planes 61-64 of FIG. 6D, the sub-mesh generation ismore complex in FIGS. 6E-6G, and each sub-mesh comprises more triangles.However, compared to a polygon generation process performed on a singleset of points comprising the four corner points of the wall plane 70,the four corner points of the first extrusion 18 and the four cornerpoints of the second extrusion 19, the polygon generation processillustrated in FIGS. 6E-6G is still less computationally demanding.

Finally, in step 237, for each wall, the sub-meshes that are generatedfrom the sub-planes in step 236 are combined to give a single mesh thatrepresents a wall of the interior space.

An extrapolation technique such as the technique illustrated in FIG. 6Dmay be preferred for walls that include extrusions which have a regularpolygon shape, as an extrapolation technique will tend to producesub-planes of a particularly simple shape. This advantageouslysimplifies the sub-mesh generation of step 236.

However, a splicing technique such as the technique illustrated in FIGS.6E-6G may be preferred for both regular and irregular extrusions. Thisis partly because it is advantageous from a memory bandwidth perspectiveto be able to use a single technique for all extrusions, regardless oftheir shape, but also because a splicing technique will tend to produceless sub-planes than an extrapolation technique. Although the shapes ofsuch sub-planes may be more complex, such that the sub-mesh generationof step 236 is more complex for each sub-plane, less sub-meshes overallhave to be generated. Further, less sub-meshes must be combined in step237.

Now returning to process 20 of FIG. 2, having generated a polygon meshfor each wall of the interior space in step 23, in step 24 the polygonmeshes representing the three-dimensional shapes of the walls areprovided to a rendering engine for creating a virtual representation ofthe interior space.

In some cases, the polygon meshes that are generated for each wall arecombined into a single polygon mesh representing the entire interiorspace before being provided to the rendering engine. Such a mesh isillustrated in FIG. 6H (it is noted that the extrapolation technique ofFIG. 6D has been used for this interior space). However, in other cases,the polygon meshes representing the individual walls are not combined,but are provided separately to the rendering engine. In other casesstill, the polygon meshes representing the individual walls are combinedinto two or more groups, each having a one or more walls, and the groupsare then separately provided to the rendering engine.

While the manner in which the meshes are provided to the renderingengine does not change the end result of the rendering, the latter twoapproaches allows a wall or a group of walls to subsequently bere-rendered without having to re-render the other walls or groups ofwalls. For example, a user may wish to make changes to one wall (a“feature wall”, for example), either at the level of the polygon mesh(the addition of an extrusion to a wall, for example) or at therendering level (a change to the surface decoration of the wall, forexample), without wishing to make changes to the other walls. The lattertwo approach permits this, without necessarily requiring computationallydemanding rendering to be performed for the entire interior space.

Described above are a number of embodiments with various optionalfeatures. It should be appreciated that, with the exception of anymutually exclusive features, any combination of one or more of theoptional features are possible.

The invention claimed is:
 1. A method for generating a virtualrepresentation of an interior space such as a room, comprising:obtaining a first set of three-dimensional coordinates and at least onefurther set of three-dimensional coordinates, wherein the first set ofthree-dimensional coordinates comprises three-dimensional coordinatesrepresenting three-dimensional positions of points located on edges ofwalls of the interior space, and wherein each of the at least onefurther set of three-dimensional coordinates comprises three-dimensionalcoordinates representing positions of points located on edges of anextrusion in one of the walls of the interior space; generating apolygon mesh representing the three-dimensional shape of the interiorspace, wherein generating the polygon mesh comprises: using the firstset of three-dimensional coordinates to determine planes representingthe walls of the interior space without considering any extrusions inthe walls; and for each wall with one or more extrusions, using therespective determined plane and the respective one or more of the atleast one further set of three-dimensional coordinates to determine aplurality of sub-meshes that in combination represent the respectivewall excluding the respective one or more extrusions; and combining theplurality of sub-meshes into a mesh representing the wall with the oneor more extrusions; and determining, for each of the at least onefurther set of three-dimensional coordinates, which of the determinedplanes the extrusion belongs to; wherein determining which of thedetermined planes the extrusion belongs to comprises comparing theorientation of a plane through the points representing positions ofpoints located on edges of the extrusion to the orientation of thedetermined planes.
 2. The method of claim 1, further comprisingnormalizing the three-dimensional coordinates of the first set ofthree-dimensional coordinates and/or the at least one further set ofthree-dimensional coordinates to account for capture drift.
 3. Themethod of claim 1, further comprising, for each wall without anyextrusions, using the corresponding plane to determine a meshrepresenting the wall.
 4. The method of claim 1, further comprisingcombining all the meshes representing all walls of the interior space togive a single mesh representing the three-dimensional shape of theinterior space.
 5. The method of claim 4, further comprising providingthe single polygon mesh representing the three-dimensional shape of theinterior space to a renderer for rendering the polygon mesh to generatea virtual representation of the interior space.
 6. The method of claim1, further comprising providing one or more polygon meshes to a rendererfor rendering the one or more polygon meshes, wherein each of the one ormore polygon meshes represents the three-dimensional shape of one ormore walls of the interior space.
 7. The method of claim 1, whereinusing the respective determined plane and the respective one or more ofthe at least one further set of three-dimensional coordinates todetermine a plurality of sub-meshes that in combination represent therespective wall excluding the respective one or more extrusionscomprises: translating or projecting the respective extrusion onto therespective plane.
 8. The method of claim 7, further comprisingparallelizing the extrusion to the plane prior to translating orprojecting the extrusion onto the plane.
 9. The method of claim 1,wherein using the respective determined plane and the respective one ormore of the at least one further set of three-dimensional coordinates todetermine a plurality of sub-meshes that in combination represent therespective wall excluding the respective one or more extrusionscomprises: dividing the plane less the one or more extrusions into aplurality of sub-planes; and generating a sub-mesh for each sub-plane.10. The method of claim 9, wherein dividing the plane less the one ormore extrusions into a plurality of sub-planes comprises at least oneof: performing an extrapolation technique; or performing a splicingtechnique.
 11. The method of claim 10, wherein the extrapolationtechnique comprises, for each of the one or more extrusions, dissectingthe plane along lines through a minimum and maximum extent of theextrusion.
 12. The method of claim 10, wherein the splicing techniquecomprises, for each of the one or more extrusions, dissecting the planethrough a central point of the extrusion.
 13. The method of claim 1,wherein the polygons are triangles.
 14. The method of claim 1, whereinthe first set of three-dimensional coordinates comprises at least oneof: a three-dimensional coordinate for each vertical edge, wherein avertical edge is an edge where two adjacent walls of the interior spacemeet; a three-dimensional coordinate for each horizontal edge, wherein ahorizontal edge is an edge where a wall of the interior space meets aceiling or floor of the interior space; or at least one height pointindicating the height of the interior space.
 15. The method of claim 1,wherein an extrusion in a wall is a door, window or fireplace in thewall.
 16. The method of claim 1, wherein obtaining the first set ofthree-dimensional coordinates comprises: displaying, on a display of anelectronic device, a live view of the interior space as captured by acamera of the electronic device; and for each of the edges, receiving auser input indicating a point on the display corresponding to the edge;converting the user input into a three-dimensional coordinate; andstoring the three-dimensional coordinate in memory of the electronicdevice.
 17. The method of claim 16, wherein an augmented reality toolkitof the electronic device is used to convert the user input into thethree-dimensional coordinate.
 18. The method of any of claim 16, whereinthe user input is converted into a three-dimensional coordinate andstored in memory as soon as the user input is received.
 19. The methodof claim 1, wherein obtaining each of the at least one further set ofthree-dimensional coordinates comprises: displaying, on a display of anelectronic device, a live view of the interior space as captured by acamera of the electronic device; and for each of the extrusions,receiving user inputs indicating points on the display corresponding tothe edges of the extrusion; converting the user input into thethree-dimensional coordinates; and storing the three dimensionalcoordinates in memory of the electronic device.
 20. The method of claim1, wherein obtaining the first set of three-dimensional coordinatesand/or the at least one further set of three-dimensional comprisesretrieving a previously captured set of three-dimensional from memory ofan electronic device.
 21. A method for generating a virtualrepresentation of an interior space such as a room, comprising:obtaining a first set of three-dimensional coordinates and at least onefurther set of three-dimensional coordinates, wherein the first set ofthree-dimensional coordinates comprises three-dimensional coordinatesrepresenting three-dimensional positions of points located on edges ofwalls of the interior space, and wherein each of the at least onefurther set of three-dimensional coordinates comprises three-dimensionalcoordinates representing positions of points located on edges of anextrusion in one of the walls of the interior space; generating apolygon mesh representing the three-dimensional shape of the interiorspace, wherein generating the polygon mesh comprises: using the firstset of three-dimensional coordinates to determine planes representingthe walls of the interior space without considering any extrusions inthe walls; and for each wall with one or more extrusions, using therespective determined plane and the respective one or more of the atleast one further set of three-dimensional coordinates to determine aplurality of sub-meshes that in combination represent the respectivewall excluding the respective one or more extrusions; and combining theplurality of sub-meshes into a mesh representing the wall with the oneor more extrusions; and normalizing the three-dimensional coordinates ofthe first set of three-dimensional coordinates and/or the at least onefurther set of three-dimensional coordinates to account for capturedrift; wherein normalizing the three-dimensional coordinates of thefirst set of three-dimensional coordinates and/or the at least onefurther set of three-dimensional coordinates comprises comparing anangle between two planes to a predetermined threshold angle, andadjusting at least one three-dimensional coordinate if the angle passesthe threshold.
 22. A non-transitory computer-readable medium storinginstructions which, when executed by a computer, cause the computer to:obtain a first set of three-dimensional coordinates and at least onefurther set of three-dimensional coordinates, wherein the first set ofthree-dimensional coordinates comprises three-dimensional coordinatesrepresenting three-dimensional positions of points located on edges ofwalls of the interior space, and wherein each of the at least onefurther set of three-dimensional coordinates comprises three-dimensionalcoordinates representing positions of points located on edges of anextrusion in one of the walls of the interior space; generate a polygonmesh representing the three-dimensional shape of the interior space,wherein generating the polygon mesh comprises: use the first set ofthree-dimensional coordinates to determine planes representing the wallsof the interior space without considering any extrusions in the walls;and for each wall with one or more extrusions, use the respectivedetermined plane and the respective one or more of the at least onefurther set of three-dimensional coordinates to determine a plurality ofsub-meshes that in combination represent the respective wall excludingthe respective one or more extrusions; and combining the plurality ofsub-meshes into a mesh representing the wall with the one or moreextrusions; and determine, for each of the at least one further set ofthree-dimensional coordinates, which of the determined planes theextrusion belongs to; wherein to determine which of the determinedplanes the extrusion belongs to comprises to compare the orientation ofa plane through the points representing positions of points located onedges of the extrusion to the orientation of the determined planes. 23.A computer system comprising one or more processors communicativelycoupled to memory and configured to: obtain a first set ofthree-dimensional coordinates and at least one further set ofthree-dimensional coordinates, wherein the first set ofthree-dimensional coordinates comprises three-dimensional coordinatesrepresenting three-dimensional positions of points located on edges ofwalls of the interior space, and wherein each of the at least onefurther set of three-dimensional coordinates comprises three-dimensionalcoordinates representing positions of points located on edges of anextrusion in one of the walls of the interior space; generate a polygonmesh representing the three-dimensional shape of the interior space,wherein generating the polygon mesh comprises: use the first set ofthree-dimensional coordinates to determine planes representing the wallsof the interior space without considering any extrusions in the walls;and for each wall with one or more extrusions, use the respectivedetermined plane and the respective one or more of the at least onefurther set of three-dimensional coordinates to determine a plurality ofsub-meshes that in combination represent the respective wall excludingthe respective one or more extrusions; and combining the plurality ofsub-meshes into a mesh representing the wall with the one or moreextrusions; and determine, for each of the at least one further set ofthree-dimensional coordinates, which of the determined planes theextrusion belongs to; wherein to determine which of the determinedplanes the extrusion belongs to comprises to compare the orientation ofa plane through the points representing positions of points located onedges of the extrusion to the orientation of the determined planes.